By Express Mail # EL489597251US 



APPLICATION FOR UNITED STATES 
LETTERS PATENT 



Semi-Transparent Handwriting Recognition UI 



Inventors: 

Minna PART AN EN 
Vesa SIMILA 



By Express Mail # EL489597251US 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0001] The present invention is related to the field of user interfaces for electronic input 

devices, and more particularly, and to a user touch-activated screen interface for a handwriting 
5 recognition system. 

2. Description of the Related Art 

q [0002] As electronic devices, such as computers, mobile telephones, PDAs and the like 

00 have become more commonplace, users expect those electronic devices to become more 

versatile and easier to use. One manner in which manufacturers have attempted to meet that 
10 h i expectation is in the form of the user interface. 

Q [0003] Manufacturers have upgraded the user interfaces of even the smallest of devices 

fIJ to include features such as color displays, and human-centered ergonomic improvements. One 
H specific improvement is in providing interfaces that permit handwritten input, by way of 

handwriting recognition. 

15 [0004] Many different systems exist for recognizing handwritten characters, and 

outputting text in response to manuscript input is taught. Examples of such systems are 
disclosed in PCT Patent document Nos. WO 96/00424 and WO 98/29791, and in U.S. Patent 
No. 6,195,638 . These systems, however, do not deal with the user interface for inputting the 
handwriting, but rather with how to decipher it once it is input. 
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[0005] A common and popular device which permits handwritten input is sold under the 

name PALM PILOT® by Palm, Inc. The PALM PILOT ® includes a dedicated touch-activated 
portion of its video display for inputting handwritten characters in a handwriting recognition 
system known as GRAFFITI®. While it is helpful for the user to have a known place that is 
5 usable for handwritten input, the dedicated area has its drawbacks. 

[0006] First, the dedicated area is of a fixed and immutable size. The user may need 

more, or less room at any given time, and a fixed-size dedicated area is not always the right 
p size. 

SO [0007] Second, the dedicated area may not be in the most useful place, as the user may 

10 c 2 Prefer for it to be in a different place, for example, a left-handed user may prefer to move the 
r!! area to a place more suitable for his input. 

Q [0008] Third, the dedicated area takes up a constant amount of space on the display, 

III even when not in use. Especially on small devices, such as a PDA, losing a significant area of 
P a user's screen for a manuscript input area when she is not using that input is bothersome. 
15 [0009] Another electronic device permitting manuscript input to a dedicated area 

established for handwriting is disclosed in U.S. Patent No. 5,049,862. This device, as well, 

however, does not address the drawbacks listed above. 

[0010] There is thus a need for a user interface for providing a more flexible user input 

for manuscript input to an electronic device. 
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SUMMARY OF THE INVENTION 

[0011] It is an object of the present invention to provide a method and system for a user 

interface for an electronic device that is more adaptable and useful than known systems. 
[0012] It is a further object of the invention to provide a method and system for 

5 providing a user interface for a handwriting recognition system for an electronic device in 
which the user inputs handwritten characters into a semi-transparent window which appears on 
a video display for such electronic device in response to the user moving a stylus across a 
Q touch-activated screen of the video display. 

II [0013] It is a still further object of the invention to provide a semi-transparent window 

10 for a user interface in a handwriting recognition system in which the semi-transparent window 
u I may be moved and sized depending upon the particular needs and preferences of the user. 

O [0014] The present invention is a method and system for a user interface of a 

m 

K handwriting recognition system intended for use in small electronic devices, such as PDAs, 
h? mobile telephones and laptop computers. However, the present invention may be used with a 
15 device of any size, including a desk-top computer, a mainframe computer, or even a consumer 
appliance such as a microwave oven. The user interface is a semi-transparent window that 
opens in response to a user-initiated manuscript input to any point on a touch-activated screen 
of a display of or associated with the electronic device. The touch-screen need not be a part of 
the device, as is the case with a PDA, mobile phone and laptop computer, but may also be 
20 merely connected, electronically or wirelessly, to the electronic device, as is the case with a 
desk-top computer. The semi-transparent window may be sized or moved, as desired by the 
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user, and may be automatically sizable in response to the placement of the user's manuscript 
input on the touch-activated screen. 

[0015] Other objects and features of the present invention will become apparent from 
the following detailed description considered in conjunction with the accompanying drawings. 
It is to be understood, however, that the drawings are designed solely for purposes of 
illustration and not as a definition of the limits of the invention, for which reference should be 
made to the appended claims. It should be further understood that the drawings are not 
necessarily drawn to scale and that, unless otherwise indicated, they are merely intended to 
conceptually illustrate the structures and procedures described herein. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0016] In the drawings : 

Figure 1 of the drawings is a screen shot of the semi-transparent window of the 

present invention in operation; 

Figure 2 is a schematic representation of the components of the invention; and 
Figure 3 is a flow chart showing the logical operation of the invention. 
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DETAILED DESCRIPTION OF THE PRESENTLY PREFERRED EMBODIMENTS 

[0017] Figure 1 depicts a video display 10, in accordance with the present invention. 

Display 10 includes a semi-transparent window 12 overlying a video display screen 14 which is 
preferably touch-activated. Semi-transparent window 12 is defined by boundaries 16, and is 
distinguished from the surrounding area of video display screen 14, for example, by boundary 
lines, or by being of a different color or brightness. The color and brightness of semi- 
transparent window 12 are selected so that the features of the underlying video display screen 
14 are still visible, thereby permitting the user to see what lies beneath semi-transparent 
window 12 yet still permitting the user also to see the contents of the overlying semi- 
transparent window 12. The user may input a manuscript character 18, for example, the letter 
"I", and that character will be recognized by handwriting recognition software in known 
fashion and output as the character "I" 20 on display 14 in the position allotted therefor. 
[0018] Figure 2 shows a schematic representation of the components of the preferred 

embodiment of the inventive system 22. Some form of data processor 24, such as that within a 
PDA, a mobile phone, computer or other electronic device, is coupled to a display 26, which 
includes a screen 28. Screen 28 is adapted to receive information from a user by being 
activated by an external input device 30. In the preferred embodiment, this activation may be 
by way of screen 28 being activated by touch, for example, by a stylus 30. Although stylus 30 
may be a device specially made for the particular touch-activated screen 28, it may also be any 
form of writing implement to input handwritten data, such as letters or numbers, to touch- 
activated screen 28. Where screen 28 is not touch-activated, input may be implemented by a 
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mouse, trackball, joystick, electronic tablet or any other form of input device. In response to 
the movement of input device 30 on screen 28, screen 28 sends a signal to data processor 24 
describing the movement of input device 30. 

[0019] Data processor 24 generates a display signal for displaying on display 26 the 

handwriting as it is generated, so that the user has some feedback on what is being written. 
This display will stay on display 26 for a predetermined period of time, .or until the user begins 
to write another character. In this fashion, the user is able to keep track of what he or she has 
written. 

[0020] The pattern of the handwritten character is relayed to handwriting recognition 

software 30 (which may be stored in data processor 24). Handwriting recognition software 30 
analyzes the user's manuscript input, and then generates a signal to data processor 24 
identifying the manuscript characters as specific letters or numbers. These characters are then 
relayed to display 26 for display to the user. 

[0021] Data processor 24 is also coupled to a data storage area 34, which may also be a 

part of the data processor's memory, for storing user preferences and other data. 
[0022] Referring to Figure 3, before commencing use of the system, a user must 

establish certain preferences, step 102, or accept the default preferences of the system. For 
example, the user must select the manner in which semi-transparent window 12 is rendered 
distinct from the background of video display screen 14. This may be accomplished by 
selecting a translucent coloring for semi-transparent screen 12, by selecting a different 
brightness therefor, etc. Other modes of differentiation may be established as well, and the 
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user must select which mode is to be used, although system defaults may require the use of a 
specific format, according to the features of the display, to provide the user with the optimal 
mode for differentiation between the appearances of semi-transparent window 12 and the 
remainder of video display screen 14. 

[0023] Another preference to be established is the size and position of semi-transparent 

window 12 upon its initial presentation on display 26. For most languages, including English, 
the screen might open so that the open area of semi-transparent window 12 is to the right of the 
opening point. For other languages which are written right to left, the opening might be on the 
opposite side of the opening point, to make it easier for the user to write in accustomed format. 
[0024] Semi-transparent window 12 also must be positioned vertically with respect to 

the opening point, and it is preferred that the position of semi-transparent window 12 relative 
to the opening touching of the input device 30 to screen 28 be selected so that the natural 
movement of the user in writing the desired manuscript character will be in the direction of the 
middle of semi-transparent window 12. Default orientations may be established depending 
upon the language of the user, whether the user is right- or left-handed, the requirements of the 
particular handwriting recognition software, etc. 

[0025] Once these preferences are established, the user may begin use of the inventive 

system in the normal course of his or her practice. 

[0026] In addition, the system may be set up so that any one or more of the pre-selected 

preferences may be changed after semi-transparent window 12 has been opened. This feature 
might be desirable, for example, if printed text displayed on the screen interferes more than 
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desired with the pre-selected location of semi-transparent window 12. Semi-transparent 
window 12 may be moved or resized in a way similar to that used in various known programs. 
For example, to resize semi-transparent window 12, the user might touch stylus 30 to a portion 
of the edge of semi-transparent window 12 and drag this edge in the desired direction, thereby 
increasing or decreasing the size of semi-transparent window 12. To move semi-transparent 
window 12, the user may touch with stylus 30 a particular place on semi-transparent window 
12 and drag the entire semi-transparent window 12 to a new position. 

[0027] To initiate the opening of the user interface, the user may simply contact touch- 

activated screen 28, step 104, at any desired location (for example at point 36 shown in Fig. 1 
where writing by the user begins). The initial contact of stylus 30 to screen 28 causes screen 
28 to generate a signal to data processor 24, which then accesses the preferences established 
for opening semi-transparent window 12, and uses those preferences to generate, step 106, 
semi-transparent window 12 on display 14. The user continues to write the manuscript 
character(s) in semi-transparent window 12, while screen 14 sends the output, step 108, 
showing the movement of stylus 30 thereon to data processor 24 which generates an output 
showing the handwritten input, step 110, as it is written, and simultaneously sends the pattern 
of the manuscript character to the handwriting recognition software 32, step 112. 
[0028] The display will stay active as determined by system preferences, for example, 

while the user is writing a single block of text, such as a single character, or word. As each 
new block of text is commenced and, identified, for example, by lifting stylus 30 from touch- 
activated screen 28, the prior display may be cleared. 
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[0029] Alternatively, the preferences may be established so that the displayed character 

18 remains displayed for a predetermined period of time, e.g., 2 seconds, after the last input 
from input device 30 to video display screen 26. 

[0030] Once the pattern is completed, handwriting recognition software 32 determines 

the intended character, and sends it, step 114, through data processor 24 to be presented on 
display 26 (see letter 20 in Fig. 1). 

[0031] In this fashion, system 10 conveniently and efficiently generates typed 

characters in response to manually generated character input. 

[0032] If the user wishes, he may move or size semi-transparent window 12, step 116. 

[0033] The display window 12 may provide for switching between capital letters and 

small letters, numbers or an international character set, for example, by touching with the 
stylus 30 the virtual buttons 38, 40 and 42 of semi-transparent window 12, Semi-transparent 
window 12 may also provide for the display of a virtual keyboard having letter keys and/or 
number keys which would allow the user to select particular letters and/or numbers by 
touching the keys with the stylus. Preferably, such a virtual keyboard would be semi- 
transparent to allow viewing of content under the window. 

[0034] Semi-transparent window 12 may also have a virtual button 44 which when 

activated will cause semi-transparent window 12 to de-activate and disappear completely. 
[0035] As a further option, certain software may provide for the automatic opening of 

semi-transparent window 12 as a prompt to the user for input to respond to a specific data 
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entry field, for example, as a response to an e-mail, as shown in Fig. 1, or in response to an 
internet web site which requires the entry of information or the completion of a form. 
[0036] Thus, while there have shown and described and pointed out fundamental novel 

features of the invention as applied to preferred embodiments thereof, it will be understood that 
various omissions and substitutions and changes in the form and details of the methods and 
devices described and illustrated, and in their operation, may be made by those skilled in the 
art without departing from the spirit of the invention. For example, it is expressly intended 
0 that all combinations of those elements and/or method steps which perform substantially the 
03 same function in substantially the same way to achieve the same results are within the scope of 
JtJ the invention. Moreover, it should be recognized that structures and/or elements and/or 
li\ method steps shown and/or described in connection with any disclosed form or embodiment of 

0 the invention may be incorporated in any other disclosed or described or suggested form or 

m 

W embodiment as a general matter of design choice. It is the intention, therefore, to be limited 
only as indicated by the scope of the claims appended hereto. 



12 



